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Music delivery concerns the trans- 
portation of music in a digital 
format to users. Music delivery 
has recently benefited from tech- 
nological progress in networking and signal pro- 
cessing. In particular, progress in networking 
transmission, compression of audio, and protec- 
tion of digital data 1 allow quick and safe delivery 
of music through networks, either the Internet or 
digital -audio broadcasting services. Additionally, 
digitalization of data makes it possible to transport 
information on content, and not just data itself, 
as exemplified by the MPEG-7 project. 2 These 
techniques give users access to huge catalogues of 
annotated music. 

These technologies address the distribution 
problem, but also raise the issue of massive 
amounts of data from which users can choose. In 
the case of music, a typical database of titles, such 
as Amazon.com or CDNow. contains about 
500.000 titles. A database containing all existing 
tonal music recordings would probably reach 4 
million titles. Including ethnic music and less 
standard types of music would double or triple 
this number. 




Goals of music selection 

The music selection problem is double sided. 
We define it from both the goals of the music lis- 
tener and those of the content provider. 

The listener's viewpoint 

The problem of choosing items is common in 
Western societies, where an ever-increasing num- 
ber of available products exist. For entertainment 
and especially music, however, the choice problem 
is specific, bec aus e the underlying goals — person- 
al enjoyment and excitement — don't fall in the 
usual categories of rational decision making. 
Although modeling a listener s goals in accessing 
music is very complex, we identify two basic ingre- 
dients: desire for repetition and desire for surprise. 

The desire for repetition is well known in 
music theory and experimental psychology. 3 - 4 At 
the melodic or rhythmic levels of music, repeti- 
tion breeds contentment. For instance, sequences 
of repeating notes create listener expectations for 
the same note to reoccur. At a higher, more com- 
plex, level, tonal music is based on structures that 
create strong expectations of events to come (for 
example. listeners expect a dominant seventh 
chord in tonal music to resolve) . At the global 
level of music selection, the desire for repetition 
leads people to listen to music they already know 
and like or that is similar to music with which 
they are familiar. 

On the other hand, the desire for surprise is a 
key to understanding music at all levels of per- 
ception. The very theories that emphasize the role 
of expectation in music also show that listeners 
don't favor expectations that are always fulfilled — 
they enjoy surprises and atypical musical progres- 
sions. 5 At a broader level, listeners want to 
occasionally discover new music, new titles, new 
bands, or new styles. 

Of course, these two desires are contradictory. 
The issue in music selection is to find the right 
compromise between providing listeners with 
titles they already know and with tides they don't 
know, but will probably like. 

The content provider's viewpoint 

From the record companies' viewpoint, the 
goal of music delivery is to better exploit the cat- 
alogue. Indeed, record companies have problems 
with the exploitation of their catalogue using 
standard distribution schemes. For technical rea- 
sons, only a small part of the catalogue is actual- 
ly active, that is, proposed to users, in the form of 
easily available products. More importantly, the 
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analysis of music sales clearly shows decreases in 
the sales of albums, and short-term policies based 
on selling lots of copies of a limited number of 
items (hits) are no longer efficient. Additionally, 
the sales of general- purpose samplers (such as a 
best-of-love-songs collection) are no longer prof- 
itable, because consumers already have the hits 
and don't want to buy CDs containing only one 
or two titles that they like. Instead of proposing a 
small number of hits to a large audience, a natur- 
al solution is to increase diversity by proposing 
more customized albums to music consumers. 

We examine the approaches to music selection 
according to these three goals: repetition, surprise, 
and exploitation of catalogues. We show that cur- 
rent approaches only partially achieve these goals. 

Approaches in music selection 

We divide current approaches in music selec- 
tion into two categories: query systems and rec- 
ommendation systems. In both cases, these 
approaches provide sets of items to the user from 
which he or she still must choose. 

The database approach 

Query systems address database issues for stor- 
ing and representing musical data. They provide a 
means of accessing musical items using some sort 
of semantic information. Users can issue various 
types of queries, either very specific (such as the 
title of the Beatles song that contains the word 
"pepper") or largely underspecified (such as jazz 
tides). In all cases, the database approach, however 
sophisticated, only satisfies the goal of repetition, 
since it provides users with exactly what they ask 
for. therefore no novelty or surprise is achieved. 

Collaborative filtering approaches 

Collaborative filtering (CF) systems 6 address 
the goal of surprise by issuing users personalized 
recommendations. CF has had some success in 
the field of music selection (Amazon.com, Firefly. 
Infoglide. and MyLaunch for example) as well as 
in other domains such as books and news. 

CF is based on the idea of patterns in tastes— that 
is, tastes are not distributed uniformly. Content 
providers can exploit these patterns by managing a 
profile for each user connected to the service. The 
profile is typically a set of associations of items to 
grade ratings. In the recommendation phase of CF. 
the system finds all agents having a similar profile 
as the users. It then searches for items prefened by 
these similar agents but aren't known to this user 
and recommends these items to him or her. 



Experimental results show that the recommen- 
dations, at least for simple profiles, are of good 
quality once the user gives a sufficient amount of 
initial ratings. 6 However, there are limitations to 
this approach, evident by studying quantitative 
simulations of CF systems using work on the dis- 
semination of cultural tastes. 7 - 8 One limitation is 
the inclination to cluster formation, which is 
induced by the very dynamics of the system. CF 
systems produce interesting recommendations for 
simple profiles, but get stuck when the profiles get- 
bigger: eclectic profiles are disadvantaged. Another 
problem, shown experimentally, is that the 
dynamics favor the creation of hits, that is. items 
liked by a large fraction of the population. If hits 
are not intrinsically bad things, they nevertheless 
limit the possibility of other items to survive in a 
world dominated by weight sums. 

CF addresses the goal of surprise in a safe way 
by proposing to users items that are similar to 
items known by them. However, cluster forma- 
tion and uneven distribution of chances for items 
(that is. hits) are the main drawbacks of the 
approach both from the user viewpoint and the 
content provider viewpoint. Users get clusters 
from which it's difficult to escape, and content 
providers don't get systematic exploitation of the 
catalogue. 

On-the-fly music program generation 

Instead of proposing sets of individual titles to 
users, we propose building full-fledged music pro- 
grams—that is, sequences of music tides—satisfy- 
ing particular properties. 

Overview 

There are two motivations for proposing music 
programs rather than unordered collections of 
tides. One is that music tides are rarely presented 
in isolation. For example. CDs, radio programs, and 
concerts are all made up of temporal sequences of 
pieces in a certain order. This order is frequentiy 
significant because different orders don't produce 
the same impressions on music listeners. The craft 
of music programming is to build coherent 
sequences rather than to just select individual titles. 

The other motivation is that properties of 
sequences play an important role in the perception 
of music. For instance, several music tides in a sim- 
ilar style convey a particular atmosphere and create 
expectations for the next titles. As a consequence, 
a listener may not particularly enjoy an individual 
title in abstract, but the tide may succeed as the 
right piece at the right time within a sequence. 



Rather than focusing on the similarities of 
individual titles, we can exploit properties of 
sequences to satisfy the three goals of music selec- 
tion. We propose building a database of titles, 
with content information for each title. We spec- 
ify sequences of music titles (music programs) by 
defining the properties or patterns we want the 
program to have. These properties are represent- 
ed as constraints, in the sense of constraint satis- 
faction techniques. Finally, a constraint solver 
computes the solutions of the corresponding com- 
binatorial pattern-generation problem. 

Working example 

The problem is to build music programs as 
temporal sequences that satisfy the three goals of 
music selection: repetition, surprise, and exploita- 
tion of catalogues. As an example, we will take a 
music program for which we specify the desired 
properties. We focus on the format of the database 
and the nature of constraints. 

Following is a liner-note description of a typi- 
cal music program. The properties of the sequence 
are grouped in three categories. This example 
describes a music program called Driving a Car, 
ideally suited for car music: 

1 . Listener preferences 

I No slow or very slow tempos 

I At least 30 percent female-type voice 

I At least 30 percent purely instrumental pieces 

I At least 40 percent brass 

I No more than 20 percent country pop style 

I One song by Harry Connick, Jr. 

2. Constraints on the coherence of the sequence 

I Styles of titles are close to their neighbors 
(successor and predecessor) to ensure some 
style continuity in the sequence 

I Authors are all different 

3. Constraints on the exploitation of the 
catalogue 

I Contains 1 2 different pieces to fit on a typ- 
ical CD or Sony MiniDisc format 



I Contains at least five titles from the label 
Epic/Sony Music, a bias to exploit the cata- 
logue in a particular region 

Database of music titles 

The database of music titles contains content 
information needed for specifying the constraints. 

Format of the database 

Each item is described by attributes taking their 
value in a predefined taxonomy. The attributes are 
technical and content. Technical attributes 
include the title, the author, the duration of the 
piece, and the recording label (for example. Learn 
to Love You, Harry Connick, Jr.. 279 seconds, 
Epic/Sony Music). Content attributes describe 
musical properties of individual titles, for exam- 
ple, style (jazz-crooner), type of voice (muffled), 
music setup (instrumental), type of instruments 
(brass), tempo (slow-fast), and other optional 
attributes such as the type of melody (consonant) , 
or the main theme of the lyrics (love). 

Currently, in our project the database is creat- 
ed by hand by experts (including coauthor 
Cazaly). However, current research focuses on 
extracting some of these attributes (such as 
tempo 9 or rhythm) automatically, when possible, 
in the context of the MPEG -7 standardization 
process. 

Taxonomies of values and similarity relations 

An important aspect of our project's database is 
that similarity relations link the values of content 
attributes. We use these similarity relations for spec- 
ifying constraints on the continuity of the sequence 
(the preceding example, Driving a Car, contains a 
constraint on the continuity of styles) . We use these 
taxonomies of attribute values to establish links of 
partial similarity between items according to a spe- 
cific dimension of musical content. 

Some of these relations are simple ordering rela- 
tions. For instance tempos take their value in the 
ordered list (fast, fast-slow, slow-fast, slow). Other 
attributes such as style take their value in full- 
fledged taxonomies. The taxonomy of styles is par- 
ticularly important because it embodies a global 
musical knowledge that the system exploits. 

Internet music retailers and others have 
designed various classifications of musical styles. 
These classifications are mainly designed for a 
query-based approach. For instance, the taxono- 
my ofAmazon.com is a tree-like classification that 
embodies a relation of generalization and special- 
ization between styles, for example, "blues" is 
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more general than 
"Memphis blues." As 
such, this type of classi- 
fication is well suited 
for navigating the cata- 
logue to find under- 
specified items, but it 
does not represent sim- 
ilarities between styles, 
for instance, having a 
common origin, like 
soul-blues and jazz- 
crooner. 

Our taxonomy of styles explicitly represents rela- 
tions of similarity between styles as a nondirected 
graph in which vertices are styles and edges express 
similarity. It currently includes 120 different styles, 
covering most of Western music (see Figure 1) . 

CSP for building music programs 

Building music programs that satisfy sets of 
constraints is a combinatorial pattern-generation 
problem. This problem cannot be solved trivially 
using a database access technique. Solving the 
sequence -generation problem using convention- 
al database techniques would require defining a 
database containing all possible sequences, and 
this is not possible. For example, the number of 
sequences of 20 items, out of a catalogue with 
100,000 possible values for each item (about the 
size of a catalogue of a major record label) is 10»°°. 

The task is in fact the opposite of pattern 
matching because in pattern matching, one looks 
for patterns in given sequences. Here, we want to 
create sequences with given patterns. 

Constraint satisfaction programming (CSP) is 
a paradigm for solving difficult combinatorial 
problems, particularly in the finite domain. In this 
paradigm, problems are represented by variables 
having a finite set of possible values, and con- 
straints represent properties that the values of 
variables should have in solutions. CSP is a pow- 
erful paradigm because it lets the user state prob- 
lems declaratively by describing a priori the 
properties of its solutions and use general-purpose 
algorithms to find them. 

Most of the CSP resolution algorithms are 
based on the same principles, where a backtrack- 
ing procedure is interlaced with a domain reduc- 
tion phase. 

The backtracking procedure. The backtrack- 
ing procedure progressively instantiates the vari- 
ables with values taken in their domains and goes 
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backward if an inconsistency is detected that pre- 
cludes finding any solution. After each step of the 
search, the algorithm performs a domain reduc- 
tion phase. 

The domain reduction phase. The goal here 
is to reduce the overall size of the remaining 
unsolved problem by removing values from the 
domains of the variables. This leads to developing 
a smaller search tree and thus speeds up the reso- 
lution. Of course, because we don't want to dis- 
card any solution, we have to remove values that 
cannot pertain to any solution. For this, we use 
constraint-filtering (also called local -consistency 
or constraint-propagation) techniques. 

Filtering a constraint means removing, from the 
domains of the variables, values that are inconsis- 
tent with the current instantiations of the problem, 
with respect to that constraint. The filtering oper- 
ation depends heavily on the nature of the con- 
straint considered. 10 The objective of CSP is to 
identify general-purpose constraints that we can 
use to specify particular classes of problems (known 
as global constraints) and design efficient filtering 
procedures for them. In effect, the efficiency of this 
general-purpose resolution scheme depends on the 
efficiency of individual filtering procedures. 

CSP for building sequences 

We formulated a music program satisfying 
constraints of a finite domain CSP where the 
sequence is composed of successive items repre- 
sented as constrained variables. The domain of 
the variables is the (finite) catalogue that is 
searched. Constraints establishing properties of 
the sequence are expressed in the CSP paradigm 
and hold on the variables and their attributes. As 
previously discussed, this formulation yields a 
hard combinatorial problem. We needed to design 
efficient filtering procedures to find solutions in a 
reasonable time. 
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Figure I. An excerpt of 
our taxonomy of 
musical styles. Links 
indicate a similarity 
relation between styles, 
such as between Jazz- 
crooner and soul-blues. 
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The community of constraint programming 
has studied constraints on sequence. For instance, 
the Sequence Constraint of the CHIP 11 commer- 
cial constraint solver enables the expression of 
complex regulation rules. This constraint is used 
to control the occurrences of some patterns in a 
sequence. Specific filtering techniques handle the 
Sequence Constraint efficiently. This constraint is 
typically used for complex timetable problems to 
specify regulation rules (for example, any employ- 
ee has a two-day rest at least twice a month). 

Another kind of sequence constraint is the 
Global Sequencing Constraint 12 of IlogSolver. 13 
This constraint is used to specify the number of 
successive items having their values in a given set. 
It's a generalization of the Global Cardinality 
Constraint 14 and is filtered by the same method. 

Our problem is different because we need to 
constrain not only the value of each item, but also 
the value of an item's attributes (such as style, 
tempo, and so on). For instance, we want to have 
five jazz music titles and three slow motion titles 
in a row. These requirements cannot be expressed 
in terms of the Sequence Constraint of CHIP nor 
of the Global Sequencing Constraint. They are * 
stated by a set of individual cardinality constraints. 

RecitalComposer 

We can-express the constraints needed to spec- 
ify music programs (listener preferences, program 
coherence, and exploitation of the catalogue) 
using a small number of global constraints: simi- 
larity constraints, difference constraints, and car- 
dinality constraints. Our resulting system, 
RecitalComposer, is composed of a constraint 
solver, a database, and associated taxonomies of 
attribute values. 



Technically, this is an extension of the well- 
known all-different constraint, for which efficient 
filtering procedures already exist. 15 

Cardinality constraints. These impose numer- 
ical restrictions on sets of items. They are the most 
difficult from a combinatorial point of view, 
because they prescribe nontrivial properties on 
the whole sequence. In our context, we identified 
two such cardinality constraints: cardinality on 
items and cardinality on attributes. 

I Cardinality on items. These specify bounds on 
items for the number of occurrences of certain 
values in the sequence. For instance, in a clas- 
sics-of-rock CD sampler, one may require 
between three and five titles by either the 
Beatles or the Rolling Stones. More precisely, a 
cardinality constraint on items is defined by an 
attribute A, a set Varof variables, a set Val of 
values, and a number range [a, b]. It requires 
that the number of variables in Var whose 
value has its attribute A in Va/be in [a, b]. 

I Cardinality on attribute values. These control the 
number of different values taken by a given 
attribute. This constraint can control the 
amount of variety in the sequence. A typical use 
for this constraint would require pieces from at 
least three different labels in a given program. 

Example. We can now express the Driving a 
Car example as a CSP on sequences, by instantiat- 
ing the global constraints defined above. 

I No slow or very slow tempos: simple unary 
constraints on each variable. 



Similarity constraints. These state that within 
a given range, the items are successively similar to 
each other with respect to the underlying tax- 
onomies. For instance, using similarity constraints, 
it's possible to state that the first 10 pieces of a pro- 
gram should have similar styles with respect to the 
classification of styles shown by Figure 1 . 

Difference constraints. These are used to ensure 
some variety in the sequence by forbidding identi- 
cal attribute values for a set of items. For instance, 
we might require that a radio broadcast doesn't fea- 
ture songs by the same singer too frequendy. This 
is achieved by stating difference constraints hold- 
ing on the author attribute and involving any suc- 
cession of. for example, 10 items in the program. 



I At least 30 percent female-type voice: cardinal- 
ity constraint on attribute "voice type." 

I At least 30 percent purely instrumental pieces: 
cardinality constraint on attribute "music 
setup." i 

I At least 40 percent brass: cardinality constraint 
on attribute "instrument." 

I No more than 20 percent country pop style: 
cardinality constraint on attribute "style." 

I One song by Harry Connick, Jr.: cardinality 
constraint on attribute "author." 




Styles of titles are close to their neighbors (suc- 
cessor and predecessor): similarity constraint 
on attribute "style." 

Authors are all different: difference constraint 
on attribute "author." 

Contains 12 different pieces: standard all- 
different constraint on variables. 



I Contains at least five titles from the label 
Epic/Sony Music: cardinality constraint on 
attribute "label.'* 

A solution to this problem appears in Table 1. 
RecitalComposer computes the solution within a 
few seconds using our global constraints and our 
constraint solver 10 on a sample catalogue con- 
taining 300 titles. 
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The simplest application 
of RecitalComposer is a 
system targeted at music 
professionals for building 
music programs from a 
given database. 



Evaluation 

We can't compare RecitalComposer to other 
systems, since we don't know of any other 
attempts at generating sequences of multimedia 
data. We can evaluate the scale-up to large cata- 
logues and the quality of results. 

Technical evaluation of the CSP approach 

We used the current RecitalComposer proto- 
type, written in Java, on a sample database of 
about 300 titles. For most problems, solutions 
were computed within a few seconds. Because we 
don't have a full database with a larger number-of 
items, we experimented on randomly generated 
databases containing up to 10,000 items. These 
experiments showed that resolution time grows 
linearly with the size of the database. However, 
those results are questionable, since the structure 
of a randomly generated database may differ from 
the structure of a real catalogue. 

Experiments on databases larger by an order of 
magnitude are in progress. We claim that such an 
increase in size is not as critical an issue as it may 
appear. The complexity of the problem depends 
more on the density of solutions in the search 
space than on the size of the search space itself. In 
our case, the density of solutions augments with 
the size of the database. Additionally, we may 
divide the database into smaller domains of inter- 
est, thus leading to a moderate increase in the size 
of the search space. 

Evaluation of resulting sequences 

The solutions found by RecitalComposer satis- 
fy two goals of music selection. Listener prefer- 
ences (repetition) are satisfied by definition. 
Exploitation of the catalogue is systematic, that is, 
no clustering or bias is introduced and therefore 



the system searches the entire database for solu- 
tions. As illustrated in the working example, spe- 
cific constraints can be added to force the system 
to exploit particular regions of the catalogue. 

Assessing the surprise goal is more difficult. 
Unknown titles may be inserted in music pro- 
grams with a high probability of acceptance by 
the listener because of the properties of continu- 
ity in the sequence. We conducted experiments to 
compare programs produced by RecitalComposer 
and programs produced by human experts 
(including coauthor Cazaly) on the same sample 
database. Results show that in ail cases, experts 
weren't able to find correct solutions and had to 
remove one or more constraints. Experts consid- 
ered the solutions found by the program good, in 
particular because it yielded items that they 
wouldn't have considered. 

Conclusion 

The technique presented here is an enabling 
technology to build music delivery services. The 
simplest application of RecitalComposer is a sys- 
tem targeted at music professionals for building 
music programs from a given database. In the 
application, the user can specify the constraints 
and launch the system on a database. The user has 
full control of all the constraints, so it's aimed at 
professionals who want to express all the proper- 
ties of the desired programs. 

PathBuilder is an application of RecitalComposer 
targeting nonprofessionals, in which the user speci- 
fies the first and last tides of the program to build. 
The system contains hidden constraints that guar- 
antee the coherence of the sequence, like continuity 
of styles and tempos. PathBuilder can, for instance, 
find a smooth path between Celine Dion's AH by 
Myself 'and Michael Jackson's Beat It In another appli- 
cation the user only specifies the stylistic structure of 
the program. Users can create a long program for par- 
ties or radio broadcasts that has an overall stylistic 
structure known in advance (such as begin with pop 
and rock, then slow songs, and so on) . 

Finally, our approach can produce music pro- 
grams in various styles by adding domain- specif- 
ic constraints. We designed and implemented a 
prototype application dedicated to Baroque 
music. We used the application to build recitals of 
Baroque harpsichord music. Recitals of Baroque 
music (seventeenth century) follow rules identi- 
fied by musicologists, while allowing a great deal 
of freedom to performers. A typical rule concern- 
ing the structure of recitals is the continuity of 
tempos between consecutive pieces. More specific 



rules are also in use, such as rules on the tonality, 
since during this period of musical history, recitals 
where allowed to modulate—change tonality 
from one piece to another — only once. Other con- 
straints concern the structure of the recital (such 
as the style of the pieces forming the introductory 
part and requiring that consecutive pieces in the 
recital are of different type). 

We envisage other applications for set-top-box 
services and digital-audio broadcasting. Our cur- 
rent work focuses on the semiautomatic creation 
and maintenance of large databases of titles. 
Indeed, we can extract some of the attributes auto- 
matically from input signals and others, such as 
similarity relations between styles, by using col- 
laborative filtering techniques. MM 
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